Model- and Exemplar-based Robust Head Pose Tracking Under Occlusion and Varying Expression

We present a modeland exemplar-based technique for head pose tracking. Because of the dynamic nature, it is not possible to represent face appearance by a single texture image. Instead, we sample the complex face appearance space by a few reference images (exemplars). By taking advantage of the rich geometric information of a 3D face model and the flexible representation provided by exemplars, our system is able to track head pose robustly under occlusion and/or varying facial expression. The system starts with a simple learning stage. The user moves his/her head with a neutral expression in front of the camera within the working space. Our system automatically builds a personalized 3D face model by fitting a generic mesh model to a near frontal facial image, and acquires a few reference images at distinct poses to sparsely sample the facial appearance space. When tracking the head under occlusion and varying expression, we match the current view against the most appropriate reference image according to the predicted pose, which is much easier and more robust than if only a single texture image is used. A robust motion segmentation algorithm is used to separate point matches corresponding to rigid head motion from those corresponding to facial deformation. The head pose can then be reliably estimated from the rigid-motion points with the help of the 3D face mesh model, even when the number of points is small. Since we use reference images during tracking, the accumulative error inherent in frame-by-frame tracking is avoided and more accurate pose estimation is achieved. We demonstrate the validity of our approach with several video sequences acquired in a casual environment.